ref(pipedream): group regions into parallel jobs per pipeline#92
Open
mchen-sentry wants to merge 16 commits intomainfrom
Open
ref(pipedream): group regions into parallel jobs per pipeline#92mchen-sentry wants to merge 16 commits intomainfrom
mchen-sentry wants to merge 16 commits intomainfrom
Conversation
Optimize env var handling so shared vars stay at stage level while region-specific vars cascade to jobs. Cache pipeline_fn results, extract get_matching_stage helper. Align s4s group to ['s4s2'] only. Add rollback-final-stage-override fixture and regenerate goldens.
…rties GoCD only supports stage-level attributes (approval, fetch_materials, etc.) with no per-job override. When regions in a group define conflicting stage properties, we now assert at build time rather than silently using the first region's values.
In Jsonnet, a missing comma between stage definitions silently merges them into a single object, causing stages to be lost. This assertion catches the issue at build time by validating that each stage object has exactly one key.
s4s is no longer a deployment group. Remove from pipeline_groups and group_order, update all test fixtures and golden files.
s4s2 is a prod region that was incorrectly removed when the s4s test group was dropped. Restore it as a single-region group in pipeline_groups.
…fix README and cleanup - Restore fetch_materials: false on both pipeline-complete stages (trigger and group pipelines) that was accidentally dropped during the rewrite - Fix README example to use s4s2 group name instead of s4s - Remove working notes from .gitignore (files already deleted) - Fix misleading comment in env-vars-precedence test fixture - Regenerate golden files
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
s4s was removed as a region (PR #91) and replaced by s4s2. Update is_st() to check for s4s2 instead of the defunct s4s region. Add s4s2 to the is-st test fixture. Also fix jsonnetfmt formatting in pipedream.libsonnet (indentation and quote style in assertion messages).
s4s no longer exists as a region, and s4s2 uses the SaaS-style multi-container deploy (not ST), so the s4s check was dead code. Simplify is_st to only match customer-* regions.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ecff2a6. Configure here.
Remove reference to specific baseline region since the baseline may not be regions[0] when regions have different stage sets.
Assert that materials, lock_behavior, and other pipeline-level attributes are consistent across all regions in a group, matching the existing stage-level property validation.
mchen-sentry
added a commit
to getsentry/eng-pipes
that referenced
this pull request
Apr 15, 2026
Pipedream v3 collapses deploy-snuba-{py,rs}-customer-{1,2,4,7} into
deploy-snuba-{py,rs}-st. Add the new st pipeline names to Slack feed
filters and Datadog region mappings. Keep old customer-* names for
backward compat during staged rollout.
Also remove dead deploy-snuba-{py,rs}-s4s entries (s4s test region was
removed in gocd-jsonnet).
Ref: getsentry/gocd-jsonnet#92
mchen-sentry
added a commit
to getsentry/eng-pipes
that referenced
this pull request
Apr 15, 2026
Pipedream v3 collapses deploy-snuba-{py,rs}-customer-{1,2,4,7} into
deploy-snuba-{py,rs}-st. Add the new st pipeline names to Slack feed
filters and Datadog region mappings. Keep old customer-* names for
backward compat during staged rollout.
Also remove dead deploy-snuba-{py,rs}-s4s entries (s4s test region was
removed in gocd-jsonnet).
Ref: getsentry/gocd-jsonnet#92
mchen-sentry
added a commit
to getsentry/snuba
that referenced
this pull request
Apr 15, 2026
Bumps gocd-jsonnet from v2.19.0 to v3.0.0 which changes the pipedream
model from one pipeline per region to one pipeline per group. The four
customer region pipelines (customer-1, -2, -4, -7) collapse into a
single deploy-snuba-{py,rs}-st pipeline with parallel jobs.
Also updates PIPELINE_FIRST_STEP from deploy-snuba-{py,rs}-s4s (no
longer exists) to deploy-snuba-{py,rs}-s4s2 (the first group pipeline).
Ref: getsentry/gocd-jsonnet#92
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds region grouping support to gocd-jsonnet. Note this change has been deployed and verified.
The pipedream deployment model will be moving from one GoCD pipeline per region to one pipeline per region group. Regions within a group run as parallel jobs in a single pipeline. The single-tenant customer regions (customer-1, customer-2, customer-4, customer-7) will be collapsed into a single
stregion groupdeploy-{service}-stpipeline with four parallel jobs.Current single-region groups like
deandusare unchanged in behavior. They produce one pipeline with one job, same as before. This change is backwards compatible and thepipeline_fn(region)callback contract is unchanged. Services still return a pipeline definition for a given region. The difference is only in how pipedream aggregates those outputs; no service template code changes are required.One item to note is that GoCD stage-level attributes (approval, fetch_materials, etc.) don't support per-job overrides, so all regions in a group must agree on these properties for a given stage. Environment variables identical across all regions stay at the stage level, while region-specific values cascade to the job level. GoCD's native precedence (job > stage > pipeline) means there's no behavioral change from the old model. In practice we don't make use of per-job environment variables, but still worth mentioning here.
This change has been verified in devinfra-example-service. Planning on a staged rollout.